Virtual Reality Environment Responsive to Predictive Route Navigation

ABSTRACT

A vehicle state sensor data record associated with a current state of a vehicle is received. A predictive vehicle navigation data record based on map data is also received. The map data is associated with a current location of the vehicle. A first element of a virtual reality environment is adapted based on the received vehicle state sensor data record, and second element of the virtual reality environment is generated based on the received predictive vehicle navigation data record.

FIELD

The following disclosure relates to generating virtual realityenvironments, and more particularly, to the creation of an in-vehiclevirtual reality environment responsive to predictive route navigation,surroundings of the vehicle, and state of the vehicle.

BACKGROUND

A need exists to make travelling by vehicle more engaging forpassengers. As semi-automated and autonomous vehicles are developed,passengers—including the driver—now have more opportunities for leisure,entertainment, and experiences while in transit. Frequently repeatedtrips such as commutes and longer trips provide an opportunity forpassengers to be entertained.

Virtual reality entertainment provides an array of games andentertainment. Conventional virtual reality entertainment is based oncreating an experience or game that may incorporate the user's input toa controller, such as a first person gaming environment. Otherconventional virtual reality entertainment includes application ofvirtual reality components while the user is in motion, such as usingvirtual reality in tandem with an amusement park ride. Amusement parkvirtual reality experiences enhance a user's experience while the useris moved through a known path with known timing. Still otherconventional virtual reality entertainment is focused on augmentedreality, applying a virtual reality visual layer to the user's field ofview. Some conventional virtual reality games allow a user wear avirtual reality device to create a gaming environment for users walkingthrough real environments, overlaying virtual reality elements in thereal world view and overlaying virtual reality elements over other usersto create a gaming environment. Such conventional games exploit realtime kinematics (RTK) in conjunction with global positioning system(GPS) information to track the user's current location. Still othervirtual reality games provide a cockpit or other simulated environmentfor a user to interact with such as flight simulators, that simulatephysical sensations such as turbulence and changing orientation andposition of the aircraft while the user navigates through the simulationusing replica controls of the aircraft.

SUMMARY

In one embodiment, a method for creating virtual reality elements usingpredictive vehicle navigation provides receiving a vehicle state sensordata record associated with a current state of a vehicle and receiving apredictive vehicle navigation data record based on map data, the mapdata associated with a current location of the vehicle. The method alsoprovides adapting a first element of a virtual reality environment basedon the received vehicle state sensor data record, and generating asecond element of the virtual reality environment based on the receivedpredictive vehicle navigation data record.

In one embodiment, a system for creating virtual reality elements usingpredictive vehicle navigation is provided. The system includes at leastone processor and at least one memory including computer program codefor one or more programs. The at least one memory and the computerprogram code configured to, with the at least one processor, cause thesystem to at least receive a vehicle state sensor data record associatedwith a current state of a vehicle receive a predictive vehiclenavigation data record based on map data. The map data is associatedwith a current location of the vehicle. The system is may also receive avehicle environment sensor data record associated with currentsurroundings of the vehicle, adapt a first element of a virtual realityenvironment based on the received vehicle state sensor data records, andgenerate a second element of the virtual reality environment based onthe received predictive vehicle navigation data record, and generate athird element of the virtual reality environment based on the receivedvehicle environment sensor data record associated with the currentsurroundings of the vehicle.

In one embodiment, a system for creating virtual reality elements usingpredictive vehicle navigation is provided. The system includes at leastone vehicle sensor, at least one map database, at least one processor,and at least one memory including computer program code for one or moreprograms. The at least one memory and the computer program codeconfigured to, with the at least one processor, cause the system to atleast receive a vehicle state sensor data record associated with acurrent state of a vehicle receive a predictive vehicle navigation datarecord based on map data. The map data is associated with a currentlocation of the vehicle. The system may also receive a vehicleenvironment sensor data record associated with current surroundings ofthe vehicle, adapt a first element of a virtual reality environmentbased on the received vehicle state sensor data records, and generate asecond element of the virtual reality environment based on the receivedpredictive vehicle navigation data record, and generate a third elementof the virtual reality environment based on the received vehicleenvironment sensor data record associated with the current surroundingsof the vehicle. The system further configured to display the virtualreality environment via the virtual reality output device.

In yet another embodiment, a non-transitory computer readable medium forcreating virtual reality elements is provided including instructionsthat when executed are operable to receive a vehicle state sensor datarecord associated with a current state of a vehicle. Instructions mayalso be included to receive a predictive vehicle navigation data recordbased on map data, the map data associated with a current location ofthe vehicle, receive a vehicle environment sensor data record associatedwith current surroundings of the vehicle, and receive an externallygenerated vehicle sensor data record from at least one other vehicle.The non-transitory computer readable medium may also includeinstructions that when executed are operable to adapt a first element ofa virtual reality environment based on the received vehicle state sensordata records, generate a second element of the virtual realityenvironment based on the received predictive vehicle navigation datarecord, and generate a third element of the virtual reality environmentbased on the received vehicle environment sensor data record associatedwith the current surroundings of the vehicle. The non-transitorycomputer readable medium may further include instructions that whenexecuted are operable to adapt a fourth element of the virtual realityenvironment based on the received externally generated vehicle sensordata record.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the disclosure are described herein withreference to the following drawings.

FIG. 1 is a flow diagram in accordance with the subject technology forcreating virtual reality elements using predictive vehicle navigation.

FIG. 2 illustrates the current location of a vehicle and its predictedroute from a top down view in accordance with the subject technology forcreating virtual reality elements using predictive vehicle navigation.

FIG. 3 illustrates the current location of the vehicle and its predictedroute from a perspective view in accordance with the subject technologyfor creating virtual reality elements using predictive vehiclenavigation.

FIG. 4A illustrates an exemplary virtual reality environment based onFIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation.

FIG. 4B illustrates another exemplary virtual reality environment basedon FIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation.

FIG. 4C illustrates another exemplary virtual reality environment basedon FIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation.

FIG. 4D illustrates another exemplary virtual reality environment basedon FIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation.

FIG. 5 illustrates an example system of the subject technology forcreating virtual reality elements using predictive vehicle navigation.

FIG. 6 illustrates an example system of the subject technology forcreating virtual reality elements using predictive vehicle navigation.

FIG. 7A illustrates an example system of the subject technology forcreating virtual reality elements using predictive vehicle navigation.

FIG. 7B illustrates an example system of the subject technology forcreating virtual reality elements using predictive vehicle navigation.

FIG. 7C illustrates an example system of the subject technology forcreating virtual reality elements using predictive vehicle navigation.

FIG. 8 illustrates an example client device of the subject technologyfor creating virtual reality elements using predictive vehiclenavigation.

FIG. 9 illustrates an example server of the subject technology forcreating virtual reality elements using predictive vehicle navigation.

DETAILED DESCRIPTION

Passengers in a vehicle experience many sensations while in-transit. Themovement of the vehicle such as direction, speed, acceleration, soundand sensory information impacts the passenger's experience. The route ofthe vehicle further contributes to the experience as the passenger willexperience sensory changes through turns, straightaways, stops, traffic,speed limit changes and other changes. The view of each passengerfurther changes as the vehicle moves through space. Other vehicles,traffic lights, geographic and manmade features of the motorist'senvironment further affect the travel experience. Navigation systemsprovide route details and contextual information. Predictive navigationsystems provide anticipatory map data and contextual information,alternate route information sensitive to traffic, and provides thevehicle with the ability to maximize mechanical efficiency of thevehicle based data associated with the road ahead.

In-vehicle virtual reality provides an opportunity to change the visualexperience of a trip, while retaining the sensory experience associatedwith the ride. Virtual reality content that is complimentary,anticipatory, reactive, and responsive to sensory information associatedwith the ride provides a more realistic virtual reality experience. Thatis, physical sensations associated with the ride may be integrated intothe virtual reality experience through the use of sensor inputs from thevehicle. Anticipatory data can further be used to create richer virtualreality content and enables the virtual reality experience to beresponsive to events and conditions beyond what the passenger cananticipate through his current field of view.

The term “vehicle” as used herein encompasses its plain and ordinarymeaning including, but not limited to a car, bus, train, plane, boat,bicycle, tramway, pedestrian locomotion. The term “vehicle” includes anymotorized or unmotorized form of transportation and/or locomotion. Thevirtual reality system may be used in tandem with an autonomous orsemi-autonomous vehicle. One or multiple passengers in the vehicle mayuse the virtual reality system. The driver may be included as one of thepassengers using the virtual reality system. “Passenger” may refer toany one or more passengers of the vehicle including the driver. Driveruse of the virtual reality system may be particularly applicable toapplications associated with semi-autonomous or autonomous vehicles. Thepassengers may each have a separate virtual reality device. Thepassengers may all share the same virtual realityexperience/environment, or one or more passengers may have similarexperiences. One or more passengers may have a differentexperience/environment. Passengers sharing similar or the sameexperiences may see and/or communicate with other passengers within thevirtual reality environment. In some game environments, passengers maywork together, or play against one another. Passengers and/or othervehicles using the virtual reality system, may also see and/orcommunicate with passengers in another vehicle via the network or viacommunication between virtual reality devices and/or vehicles. Sharedexperiences/environments may reduce processing demands by receiving andcomputing data at a common location and provided to participating clientdevices.

A goal of the disclosure is to provide a virtual reality device combinedwith vehicle sensor data and predictive navigation data that correspondsto and is reflective of a dynamic environment of the vehicle. Vehicletravel may be made more engaging for passengers. The dynamic environmentmay include recognition of a recurring route, such as commuting home orto work and further includes routes with destinations that are unknownto the virtual reality system. A further goal of the disclosure is togenerate a virtual reality environment using elements corresponding tosensations that the passenger will feel in the vehicle. A goal of thedisclosure is to provide a smooth transition between the virtualenvironment and the physical world. Indications and/or messages may beprovided to the passenger indicating that the virtual realityenvironment will end. The subject disclosure departs from conventionalsolutions in that the virtual reality environment is based on an unknownroute. The subject disclosure presents an environment that is reflectiveof the vehicle's surroundings in a dynamic way. Unlike other virtualreality technology, the virtual reality system is not complete beforecommencing a trip. Thus, the virtual reality experience is not based ona static, repeating scenario. In contrast, the virtual realityenvironment of the subject disclosure adapts to the actual drivingexperience.

Elements of a virtual reality environment include, but are not limitedto brightness, color, contrast, content, sound effects, dialog, music,white noise, foreground visual elements, background visual elements,speed of virtual motion, and perspective view elements. Elements of thevirtual reality environment may further include components of stories,games, characters, and locations. Elements of the virtual reality mayfurther include scenarios including one or more other elements of avirtual reality environment.

FIG. 1 is a flow chart of a method of the subject technology forcreating virtual reality elements using predictive vehicle navigationthat may be performed by server 125 to generate and/or enhance a virtualreality environment. Some or all of the acts may be performed byprocessor 300 of server 125. Alternately, some or all of the acts may beperformed by a processor 200 of a client device such as mobile device122, vehicle 129, or virtual reality device 133. Different acts of themethod may be performed by multiple processors. Portions of the methodmay be performed by processors 200 of client devices 122, 129, and/or133. Portions of the method may be performed by processors 200 of clientdevices 122, 129, and/or 133 only when server 125 is unavailable vianetwork 127 (e.g., when the vehicle is in a remote area, underground, ina tunnel, in an enclosed structure such as a parking garage). Processors200 of client devices 122, 129, and/or 133 may alternatively performportions of the method based on processing requirements, if a route isknown before commencing the route, if a trip is known, if a trip isknown to be of a short duration, if a connection to network 127 is lost,or the like. Acts of FIG. 1 are described with reference to the systemand components depicted in FIGS. 5-9. Additional, different, or feweracts may be performed. Acts may be performed in orders other than thosepresented herein or may be performed simultaneously or substantiallysimultaneously. The term “substantially simultaneously” as used hereinrefers to timing of events temporally so close together as to beconsidered or perceived as being simultaneously occurring, for example,the time of receipt or transmission of data that is too close to measureor otherwise discriminate among.

In act S101, server 125 receives a vehicle state sensor data recordassociated with a current state of a vehicle. Vehicle state sensor datarecords may include or comprise speed, acceleration or deceleration,engine pressure, engine temperature, gear shifting, tire pressure, wheelspeed and vibration, windshield wiper setting, turn signal setting, gearselection, headlight and interior light settings, window sensors,in-vehicle climate settings, obstacle detection, Adaptive Cruise Control(ACC), autopilot steering control, safety features, autonomous drivingsystems and/or navigation system of the vehicle. Vehicle state sensordata records may additionally include any other sensors 126 associatedwith mechanical and electrical components of vehicle 129. For example,when the driver (or an autonomous vehicle) brakes, the vehicle 129 willdecelerate and the vehicle 129 will be in a state of deceleration.

In act S103, server 125 receives a predictive vehicle navigation datarecord based on map data, the map data associated with a currentlocation of the vehicle. Predictive vehicle navigation data records mayprovide data associated with road conditions and elements that arewithin a range of several kilometers/miles around the current locationof vehicle 129. Predictive vehicle navigation data may include dataobtained via sensors 126 of vehicle 129 in addition to data from sourcesphysically separate from vehicle 129. Map data, including, but notlimited to, high definition (HD) map data may be stored in databases 123a-n associated with electronic horizon generator 131. HD map data mayinclude imagery obtained using Light Detection And Ranging technology(LiDAR). Predictive vehicle navigation data may further include dataoriginating from other vehicles 129, data originating or stored onnetworking devices 135, and/or data originating from traffic devices137. Predictive vehicle navigation data may be associated with traffic,accidents, and/or geographic areas that are at a distance beyond whatcan be seen by passengers in vehicle 129.

In one example, audio is captured by vehicles ahead of vehicle 129.Construction noises, traffic, emergency vehicle sirens, or other audio.In some examples a decibel level may be identified to determine how loudor quiet the upcoming road is compared to the current location of thevehicle.

In act S105, server 125 receives a vehicle environment sensor datarecord associated with current surroundings of the vehicle. Vehicleenvironment sensor data may be obtained by short range sensors 126integrated or coupled with vehicle 129, providing data associated arange up to several hundred meters/feet. Short range sensors 126 mayinclude radar data. The vehicle environment may be identified as whetherthe vehicle is on a slope. The slope detection may be comprised ofcombinations of sensor data, such as acceleration and braking indicatingthat the vehicle is on a downward slope. The vehicle 129 sensors mayidentify nearby vehicles and detect the current speed of those vehicles.The vehicle environment may further identify changing light conditions,such as night fall, or the vehicle's entry into a tunnel or parkinggarage.

In act S107, the server 125 receives an infrastructure data record fromroadway infrastructure. Roadway infrastructure communication may includedata between vehicles and roadway infrastructure. Roadway infrastructurecommunication may also include communication with traffic devices 137including traffic signal phase data and timing information. Based on thetraffic signal phase information, it may be determined whether thevehicle 129 will have to stop or not at a traffic light. Roadwayinfrastructure data may further identify whether bridges are open orclosed, tolling stations, parking lot information (including open spaceinformation), accident data, weather or road conditions, and the like.Receipt of roadway infrastructure data may provide accident or trafficdata before such information is available through the vehicle's sensors.

The server 125 receives an externally generated vehicle state sensordata record from at least one other vehicle in act S109. Other vehiclesproximate to the current location of vehicle 129 may additionallyprovide information that may be used in the virtual reality environment.Current vehicle state information gathered by other vehicles 129 may beprovided to vehicle 129 such as braking and acceleration information,turn signals, or other vehicle instrument sensors. Passengers in nearbyvehicles also using the virtual reality environment may share anexperience or may be interactive within a threshold distance. Externalvehicle information such as turn single indications may be used toanticipate whether another vehicle is entering the lane. Emergencyinformation, such as a nearby vehicle sliding or hydroplaning on theroad may be detected or provided. Precomputed information associatedwith electronic horizon generator 131 may be passed to vehicle 129 fromother vehicles.

In act S111, server 125 adapts a first element of a virtual realityenvironment based on the received vehicle state sensor data record. Thefirst element of the virtual reality environment is adapted to provide acomplementary experience to the current state of the vehicle. Multiplereceived vehicle state sensor data records may be interpreted incombination to deduce the current vehicle state.

For example, deceleration of the vehicle may constitute the receivedvehicle state sensor data record and the corresponding rate at which thevirtual reality environment changes may be reduced in proportion withthe degree of deceleration. The deceleration of the vehicle may bedetected by an accelerometer or via sensors associated with the brakingsystem of vehicle 129. The adaptation of the virtual reality element ofrate may correspond such that the background of the virtual realityenvironment progresses at what appears to be the same rate as imageswould appear through the windshield of vehicle 129. In an example of avirtual city tour, virtual travel through the city scene may appear todecelerate at the same rate as vehicle 129. Additional virtual realityelements, such as the level of detail or sharpness of focus the virtualreality environment may be increased corresponding to deceleration. Inthe virtual city tour example, informational text may appear identifyinglandmarks, providing trivia, history, and the like. More, or moredetailed informational text may appear as a result of the receivedvehicle state of deceleration of vehicle 129.

If the vehicle state sensor data record indicates that the vehicle is inlow speed traffic or has stopped moving, the virtual reality elementsmay be adapted accordingly. In an example of a historical or tourismtheme, additional textual or audio information may be provided based onthe nearby landmarks. The environment theme may change to a quieterenvironment, such as a placid lake or pastoral scene. The amount ofvisual information or the resolution of the virtual environment may beadjusted. The first person perspective may change to a third person viewor birds eye view while the vehicle 129 is idling.

If the vehicle state sensor data record may indicate whether ACC hasbeen engaged. Once engaged, the vehicle state sensor data record mayfurther include sensor data associated with the current and/or futurespeed of the vehicle 129 based on maintaining a threshold distance fromother vehicles. Virtual reality elements may then be based on the ACCdata records to provide an advanced indication of changes in the speedof vehicle 129. For example, the generated virtual reality elementsprovide changes in the speed of play in the virtual world to correspondwith anticipated speeds of the vehicle received via ACC data records.The resolution of the virtual environment elements may increase based onanticipated reductions in speed, or new scenarios, change in theme, oraddition of plot points adapting the speed of play in the virtual worldmay be provided. Gear shifting sensor data may be combined with steeringsensor data to determine whether the vehicle is, or is about to changegear, reverse direction, or make a u-turn. The vehicle information maybe used to determine virtual reality elements such as field of view.Other virtual reality elements, providing virtual reality elementscorresponding to the sensation of moving backwards (e.g., a gust ofwind) and/or centripetal force (e.g., a merry-go-round), may be providedupon indication of different vehicle maneuvers.

Data associated with windows opening, or climate control changes, suchas turning on the air-conditioner initiated by passengers within vehicle129, may be used to generate virtual reality elements, such as a windowbreaking in a space shuttle environment, a snow storm, gusts of wind orother generated events and objects.

In act S113, server 125 generates a second element of the virtualreality environment based on the received predictive vehicle navigationdata record. A predicted route associated with the predictive vehiclenavigation data records may correspond to a path in the virtual realityenvironment. Predictive vehicle navigation data records may be createdor processed by electronic horizon generator 131 and provided to virtualreality generator 121.

Electronic horizon information may include topographical informationsuch as skyline or building outline information associated with HD maps.Such information obtained and processed by the virtual reality generator121 such that the virtual reality content is generated having a similarprofile, outline, or appearance of the physical objects and backgroundencountered by the vehicle 129. Obtaining such predictive vehiclenavigation data provides a dynamic virtual environment based on theactual course of the vehicle, even when driver's intended route and/ordestination is unknown by the virtual reality generator 121 and/orelectronic horizon generator 131.

Electronic horizon information may include number of lanes, slope, typeof road (such as dirt, gravel, paved, or brick), type of surface,curvature, possible intersections, possible traffic ahead, trafficlights. Electronic horizon information may be able to predict speedchanges (acceleration or deceleration), lane changes, route, or otherelements that may be implemented with corresponding elements in thevirtual reality environment. Predictive navigation data may includeanticipation of traffic conditions ahead such as traffic jams, atransition from interstate driving to city driving, construction,accidents and other traffic conditions that will affect the length ofthe trip.

The receipt of road environment information at further distances mayimprove the available resolution, level of detail in the virtual realityenvironment. Data processing may begin in advance of the arrival of thevehicle to the location corresponding to virtual reality environmentcomponents, preventing gaps in the virtual reality experience withoutadvance knowledge of the route.

Predictive vehicle navigation data may include audio information, audioelements of the virtual reality environment may be generated, augmented,or enhanced. The virtual reality audio level may be augmented byincreasing or decreasing a volume of the virtual reality audio levelwhen the current location of the vehicle 129 reaches the location atwhich the audio information was collected by the vehicles ahead ofvehicle 129. Visual virtual reality content may additionally bealternatively or additionally generated based on the predictive audio.If it is known that loud construction noises will likely be encounteredby vehicle 129, the computed virtual world may include backgroundvisuals such as explosions or heavy machinery calculated to appear asthe vehicle physically encounters the construction area. Virtual realityevents may be also be generated to complement the sound. Virtual realityelements may be generated to have an inversely proportional relationshipto the predictive audio information. That is, white noise levels may beincreased through the virtual reality device 133 to reduce the effect ofloud noises outside the vehicle, or the sound effects and/or musicplaying in the virtual reality environment may be increased or decreasedso that the audio level of the virtual reality environment appears toremain constant or remains audible regardless of the sounds outside ofvehicle 129. The virtual reality environment may mitigate extremechanges in sound level encountered by the passengers. That is, thevirtual reality volume may be augmented to gradually increase anddecrease (or decrease and increase) as the vehicle approaches and exitsthe physical environment with loud noise. Audio portions of the virtualreality environment may be played through the virtual reality device133, mobile device 122, and/or vehicle 129.

The electronic horizon generator 131 may receive information fromvehicle 129 regarding the location and speeds of other vehicles that arenearby. The electronic horizon generator 131 may predictively determinewhether vehicle 129 will overtake or pass any of the nearby vehicles andpredict increases in speed and lane changes based on the information.These predictive indications may be translated into side to sidemovement and increases or decreases in speed within the virtualenvironment as well. Objects may be created in the virtual realityenvironment that approximate the location and size of the nearbyvehicles in the virtual environment. In a game-type virtual realityenvironment, the events in the game may be generated to correspond orcompliment the current surroundings of the vehicle 129. Predictivenavigation data indicating that the speed of the vehicle will be reducedthrough traffic congestion ahead, the generated virtual reality elementsmay include a new scenario such as an attack of monsters in a gamingenvironment, change in theme, or addition of plot points to justify thespeed of play in the virtual world.

In act S115, the server 125 generates a third element of the virtualreality environment based on the received vehicle environment sensordata record associated with the current surroundings of the vehicle.Upon detection of environment sensor data record of the current speed ofvehicles nearby, elements in similar spatial relation to those externalvehicles 129 may be created and displayed in the virtual environment.The detection of objects, such as other vehicles, and the state of thoseobjects may dictate the movement or spatial relation of objects in thevirtual reality environment. For example, other vehicles may berepresented as animals, landscape objects, game characters, or otherobjects that may move in correspondence with the current surroundings ofvehicle 129. Changing light conditions of the environment of vehicle 129may have a darkening or lightening effect in the virtual realityenvironment, or may result in changing the virtual reality environment,such as entry into a cave corresponding with entry into a tunnel byvehicle 129. The generated virtual reality environment may includechanging the opacity of a portion or all of the display of the virtualreality device, such that elements in the physical environment areunobstructed from the passenger's view. For example, if police vehicleis identified (e.g., via image recognition associated with the vehicle'scamera or other sensor) and the gaming environment includes a bankrobbery scene, portions of the virtual reality display device 133 maybecome see-through, such that the view of the physical police vehicle isincorporated into the virtual reality environment. Opacity of thevirtual reality device 133 may be controlled as a mask layer, segmentingthe desired visual element from the entire field of view, while theidentified object (e.g., the police vehicle) is detected by the sensors126 of vehicle 129 and within the field of view.

The server 125 adapts a fourth element of the virtual realityenvironment based on the received externally generated vehicle sensordata record in step S117. The vehicle 129 may receive externallygenerated data transmitted from other vehicles 129 nearby. The state ofexternal vehicles may also provide anticipatory information, such asdata indicating that a vehicle ahead has turned on its left turn signalindicating that the vehicle ahead will turn left. This anticipatory datamay be used to pre-process virtual reality elements, such as themovement, appearance, and size of a corresponding virtual realityelement.

The server 125 adapts a fifth element of the virtual reality environmentbased on the received infrastructure data record in act S119. Roadwayinfrastructure information may provide information such as whether thevehicle 129 will be stopped by the traffic signal ahead, relating to thespeed of play, or movement of content in the virtual realityenvironment. Information such as the condition of bridges or parking lotinformation may relate to the size, shape, and layout of objects withinthe virtual reality environment. Weather information may relate tofilters applied to the virtual reality environment creating brighter ordarker environments or may result in weather based objects or backgroundelements in the virtual environment.

In act S121, the server sends an instruction to adjust an instrument ofthe vehicle based on the virtual reality environment. The virtualreality environment may further influence the physical area around thepassenger. There may be haptic sensors in the seat. The seat position,climate, position of sunroof, or other instruments of the vehicle may bealtered to enhance the virtual experience. For example, a seat warmermay be activated or deactivated or the position of the seat may bechanged based on the virtual reality environment. The virtual realitysystem may further request passenger input to impact the virtual realitysystem and/or the car's motion itself. Small changes in vehicle speed orbraking of the vehicle may be controlled based on events in the virtualreality environment, if such changes can be safely made based on thevehicle's current position, state, and environment.

The server 125 sends instructions to client devices 122, 129, and/or 133to display the virtual reality environment via the virtual realityoutput device in act S123. The virtual reality environment output devicemay include displays on mobile device 122, virtual reality device 133,and or vehicle 129, or a combination thereof.

Vehicle state data, predictive navigation data, vehicle environment mayindividually or collectively provide an indication that of the end ofthe trip. At the end of the trip, elements of the virtual realityenvironment may provide visual, physical, or aural signal to thepassenger. In semi-autonomous or autonomous vehicle applications,elements of the virtual reality environment of the driver may provideadvance indication that the driver will need to take control of thevehicle. A countdown may be provided. For example, a countdown of oneminute prior to returning control to the driver may be provided. Thevirtual reality environment may provide another indication that thevirtual reality environment is ending. For example, in an autonomous orsemi-autonomous car, the video feed of the virtual reality environmentmay indicate to the passenger that the passenger needs to take controlof the vehicle. Elements of the environment may be adjusted, such as theending of a game, completion of a level, end of tour, pause, opacity ofthe virtual images may decrease, or other virtual end may be generated.The virtual reality environment may automatically end upon detection ofan impact (such as being struck by another car), proximity of anemergency vehicle within a threshold distance. Emergency messages notnecessitating the end of the virtual reality environment may be overlaidor provided within the virtual environment. An option may be providedunder some circumstances asking the passenger whether she would like toend the virtual environment based on events or sensor data. Thesetransition elements provide the passenger with a smooth transition outof the virtual reality environment and prepare the passenger to exit thesimulation and/or to take control of the vehicle.

Additional transition elements may provide the passenger with theability to interact with other passengers in vehicle 129 ormessages/calls received via the passenger's mobile device 122. Apassenger in vehicle 129 may interact with vehicle 129 to provide aninput indicating that communication is desired. A notification may beprovided via the virtual reality device 133. The notification, such as asound or visual indication may be timed based on events in the virtualreality system, such as at the end of a game battle, end of level, orother suitable time for an interruption within the virtual environment.A notification may also be provided to passengers in the vehicle (viavehicle 129 or virtual reality device 133) once the notification hasbeen provided in the virtual reality environment, or once the anappropriate time for interruption has been determined.

FIG. 2 illustrates the current location of a vehicle and its predictedroute from a top down view in accordance with the subject technology forcreating virtual reality elements using predictive vehicle navigation.The current location of vehicle 129 in a simplified birds eye view maybe represented by black circle 129. The vehicle's location may bedetermined relative to map data such as map 250. Through predictivenavigation/electronic horizon information, a predicted route 252 may bedetermined by electronic horizon generator 131. Features of the area,such as buildings 254, 256, 258, and 260 and roads 262, 264, and 266 areillustrated in FIG. 2. Vehicle 129 is located on road 264, with building258 in the direct line of sight of the passengers.

FIG. 3 illustrates the current location of the vehicle and its predictedroute from a perspective view in accordance with the subject technologyfor creating virtual reality elements using predictive vehiclenavigation. FIG. 3 corresponds to the first person perspective view ofthe passengers of vehicle 129 as located in FIG. 2. As with first personperspective, movement is represented from this perspective via changesin the background, mid ground, and foreground elements maintaining asimilar perspective as the passenger traverses through the environment.None or some of the passenger may be visible within the frame. As avehicle 129 proceeds, the vehicle may move along the predicted route 252or depart from the predicted route, the perspective remains constant andthe passengers and immediate surroundings may remain illustrated fromthe first person perspective. Some virtual reality environments may usesome or all of the physical environment adding virtual reality elementsas layers. For example, a virtual reality environment may be generatedbased on HD map data stored and electronic horizon generator 131. Streetimages may be obtained from the databases 123 a-n of the electronichorizon generator 131 and combined to form a realistic or semi-realisticimage of the vehicle's physical environment. Virtual reality elementsmay be added, or the virtual reality environment may be skewed toprovide a different experience based on the state and/or environment ofthe vehicle and predictive navigation data acquired by vehicle 129. Inone example, the environment may generally correspond with the currentphysical environment of vehicle 129; however, the perspective of thepassenger may be raised in the virtual reality environment such that thevirtual environment appears as if the passenger is flying, rather thangrounded. In this flying example, speed, incline, decline, acceleration,and deceleration of vehicle 129 may change the perspective height of thevirtual flight and speed of the virtual reality environment.

Some or all of the actual, physical view of the passenger may beunobstructed. Virtual reality device 133 may include portions of clearor variable opacity displays that provide the passenger with a partial(or total) view of the physical environment with virtual realityelements projected onto the display. The virtual reality device 133 maynot cover the entire field of view of the passenger.

FIG. 4A illustrates an exemplary virtual reality environment based onFIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation. Theillustrated environment is representative of a rural landscape. Therural landscape may correspond to a geographical area that has featuresapproximately corresponding to the features of the current location ofvehicle 129. Features such as barn 254, windmill 256, hay bales 258, andtrees 260 correspond to the buildings 254, 256, 258, and 260 surroundingthe vehicle 129 in the physical world (as illustrated in FIG. 3). Dirtroads 262, 264, and 266 correspond to the roads 262, 264, and 266immediately surrounding vehicle 129. The virtual reality background maybe based on image data associated with a rural landscape with only somecorresponding features. For example, LiDAR imagery of a small farm maycorrespond to dirt roads 262, 264, and 266 and barn 254. As used herein,the term “roughly corresponds” and “generally corresponds” encompasstheir plain and ordinary meaning including the paths that share amajority of the same turns, straightaways and layout. “Roughlycorresponds” and “generally corresponds” may also mean that less thanall turns are represented in one path but not in another withcorrespondence between a majority of the illustrated paths. The width,angle between two connected paths, number of lanes, traffic signals, andother details between paths may differ and still maintain a “roughcorrespondence” or “general correspondence.” These image elements may becombined with generated virtual reality elements for correspondence tothe physical environment of the vehicle. That is, the virtual realityenvironment may include components taken from map imagery stored inelectronic horizon generator 131 and combined with generated virtualreality elements in order to correspond with the vehicle's currentlocation. Alternatively, some or all of the virtual reality environmentand maybe rendered or stylized based on existing images in map databasesand enhanced through virtual reality by virtual reality generator 121.

The presented virtual reality environment may include areas or citiesremote from the vehicle's current location. For example, a selection ofcities may be provided, with rough correspondence between the currentpath of the vehicle and the perspective view of the cityscape. Forexample, as the vehicle turns in physical space, the virtual realityenvironment may also turn onto the next available or path that roughlycorresponds with the path taken by the vehicle in physical space.

FIG. 4B illustrates another exemplary virtual reality environment basedon FIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation. Theillustrated environment depicts a space theme in which planets 254, 256,258, and 260 correspond to the similarly numbered buildings of FIGS. 2and 3. The paths between the planets correspond to the generally pathsof roads 262, 264, and 266 and are similarly labeled in FIG. 4B. Thespace environment may be selected by virtual reality generator 121 basedon the current speed of the vehicle or based on predictive navigation.For example, if the current state of vehicle 129 is a speed indicatingvehicle 129 is on a highway, interstate, or Autobahn (relative highspeed travel for vehicle 129), the virtual reality environment may beselected as the space environment because the vehicle is at a relativehigh speed. When virtual reality generator 121 determines via electronichorizon generator 131 that the vehicle will accelerate from the relativelow speed (city driving or traffic jam) to the relative high speed ofinterstate driving with no traffic, the current scene in the virtualreality environment may change from one environment, such as the rurallandscape (which may be used as an environment associated with lowerrelative speeds of vehicle 129) to the space scene depicted FIG. 4B.

FIG. 4C illustrates another exemplary virtual reality environment basedon FIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation. Anunderground cave exploration virtual reality environment is illustratedin FIG. 4C. Path 264, tunnel 266, and river 262 of the virtual realityenvironment may roughly correspond with roads 262, 264, and 266 of FIGS.2 and 3. The general topography of the physical environment of FIG. 3 isgenerally repeated in the underground cave exploration environment;however, the structures present in the physical world are replaced withrock formations such as rock walls 256 and 260, stalagmites 254, andstalactites 158. Elements in the virtual reality environment may begenerated based on predictive navigation data determined by electricelectronic horizon generator 131. For example, the predictive navigationdata may anticipate that the vehicle will accelerate from the currentvehicle state upon turning on road 262 of predicted route 252. Based onthe anticipated change in state the virtual reality environmentelements, paths 264 and 262 may be generated accordingly. For example,at relative low speeds, path 264 may appear as a dirt tunnel andrelative high speed up travel on path 262 may be rendered as anunderground river with strong currents. Predictive navigation data mayalso include anticipated travel speed data for alternate or availablepaths for vehicle 129 in addition to the predicted path 252. Thispredictive alternative path data may be used to generate elements in thevirtual reality environment that will be illustrated within the frameeven if the passenger and the vehicle 129 continue along the predictedpath 252 in the physical world. Virtual reality elements for alternativepaths outside of the current field of view of the passenger may begenerated and stored by virtual reality generator. If vehicle 129departs from the predicted path 252, virtual reality environmentelements pre-generated and stored alternative path virtual reality maybe used. If vehicle 129 departs from the predicted route 252, electronichorizon generator 131 may determine a new predicted route based on thecurrent position of vehicle 129. Electronic horizon generator 131 maycontinually update predicted route information based on the currentlocation of the vehicle. Virtual reality elements for predicted routesmay be generated before the vehicle 129 physically encounters thoseportions of the path such that the virtual reality environment and/orelements may be generated and transmitted over network 127 from virtualreality generator 121 to mobile device 122, virtual reality device 133,and/or vehicle 129 before the data is needed to render the display forthe passenger's experience. Accordingly, a robust virtual realityexperience is created because unanticipated losses of network 127 maynot affect the virtual reality environment displayed on virtual realitydevice 133, because one or more of the client devices may be capable ofprocessing and/or rendering and displaying virtual reality elements andenvironments previously received over network 127. Preprocessing andsending of virtual reality elements and/or environments may prevent gapsin the virtual reality experience due to network connectivity. Becausepredictive vehicle navigation may identify areas in which networkconnectivity will be lost well in advance of the vehicle physicallyapproaching an area, the virtual reality environment elements may beregenerated for the length of the anticipated outage and transmittedover network 127 to client devices before the outage will occur,providing a benefit of the disclosure.

FIG. 4D illustrates another exemplary virtual reality environment basedon FIGS. 2 and 3 in accordance with the subject technology for creatingvirtual reality elements using predictive vehicle navigation. FIG. 4D isrepresentative of a virtual reality gaming environment that may begenerated for interactive virtual reality experiences. The passenger mayinteract with the virtual reality device 133, mobile device 122, and/orvehicle 129 to provide inputs to the virtual reality game. Gameenvironment elements 254, 256, 258, and 260 generally correspond to thebuildings of FIGS. 2 and 3. Similarly, paths 262, 264, and 266correspond to the paths that are similarly numbered roads of FIGS. 2 and3. Foreground element rainbow 254, waterfall 258, lollypop 256, andbrick wall 260 may change in visual appearance based on detected oranticipated changes in the environment of vehicle 129.

Additional themes may be presented in virtual reality environments ofthe subject disclosure. The virtual reality experience may enhance theexisting view of the physical world by providing historical or tourisminformation associated with landmarks in the vicinity of the vehicle.Other gaming environments and applications are also contemplated. Thevirtual environment may be a jungle, waterway, safari, desert, or anyother scenario that compliments the sensations of the vehicle. One thememay be used throughout the virtual reality experience, or themes may beselected based on the vehicle's physical trip. For example, the thememay switch from a space theme to a jungle based when the car goes from arelative high speed to a relative low speed.

FIGS. 5-9 illustrates various example systems 120 its components forcreating virtual reality elements using predictive vehicle navigation.As illustrated in FIGS. 5-7C, multiple variations of systemarchitectures possible without departing from the scope of thedisclosure. In FIG. 5, system 120 may include a virtual realitygenerator system 121, one or more client devices (such as mobile device122, vehicle 129, or virtual reality device 133), one or moreworkstations 128, and a network 127. Additional, different, or fewercomponents may be provided. For example, many client devices 122, 129,133 and/or workstations 128 may be connected with the network 127. Thevirtual reality generator system 121 includes a server 125 and one ormore databases. The server 125 may maintain multiple databases 123 a,123 b . . . 123 n. The term database and refers to a set of data storedin a storage medium and may not necessarily reflect specific anyrequirements as to the relational organization of the data. The term“server” is used herein to collectively include the computing devices atthe virtual reality generator system 121 for creating, maintaining, andupdating the multiple databases 123 a-n. Any computing device may besubstituted for the client device 122, 139, and 133. The computingdevice may be a host for a website or web service such as a virtualreality service, mapping service, a navigation service, or a predictivevehicle navigation service. A mapping service may provide maps generatedfrom the databases 123 a-n using geographic data, and the navigationservice may calculate routing or other directions from the geographicdata. A predictive mapping service may include or access a mappingservice and/or navigation service to provide anticipatory mappinginformation. A virtual reality service may provide virtual realityelements and/or environments based on information received from clientdevices, mapping services, navigation services, and/or predictivevehicle navigation services of the databases 123 a-n.

Client devices such as mobile device 122, virtual reality device 133,and/or vehicle 129 may include processors and databases providing someor all of the services provided by virtual reality generator 121 and/orelectronic horizon generator 131. Local virtual reality generator 121and/or electronic horizon generator 131 of the client devices mayoperate in tandem with remote virtual reality generator 121 and/orelectronic horizon generator 131 via network 127. As described herein,processing of virtual reality environment and elements as described withreference to virtual reality generator 121 additionally, alternatively,or jointly may be performed by virtual reality generators disposed in orintegral with mobile device 122, virtual reality device 133, and/orvehicle 129.

Passengers may further provide information such as navigationinformation, destination, start, length of trip or other information viaclient devices. This information may be used by electronic horizongenerator 131 and/or virtual reality generator 121 to reduce processingneeds. Trips with user entered route information, such as a destination,may result in reduced computational requirements on electronic horizongenerator 131 and virtual reality generator 121, because elements of thetrip are known and the necessary predictive navigation data may bereduced. The virtual reality generator 121 and electronic horizongenerator 131 may further employ adaptive learning techniques todetermine frequently repeated routes, destinations, and tendenciesassociated with drivers and other passengers. One example of a repeatedroute is a commute to the driver's workplace. The virtual realitygenerator 121 and/or electronic horizon generator 131 may identify thisrepeated route and store or pre-compute elements of the virtual realityenvironment in association with the route and/or time of travel.Repeated routes or portions of repeated routes may be stored locally onclient devices, requiring reduced or no interaction with virtual realitygenerator 121 and/or electronic horizon generator 131 via network 127.If vehicle 129 begins a repeated route and departs from the repeatedroute, new virtual reality information may be generated based on thechanging route. A repeated route or a virtual reality experienced thatis favored by the passenger may be identified by the passenger orautomatically identified via the virtual reality generator 121 and maybe stored on databases of the mobile device 122, virtual reality device133, or vehicle 129 for ease of access.

The use of the electronic horizon generator 131 and virtual realitygenerator 121 over network 127 may provide the ability to identifylocation and conditions of the vehicle and vehicle's surrounding withina range of several kilometers/miles from the current location of vehicle129. Based on this advance knowledge of possible vehicle information,the virtual reality environment may be generated and cached in a clientdevice 122, 133, or 129, avoiding connectivity issues with network 127,and providing additional processing time to create more elaborate ordetailed environments reflective of the location of vehicle 129 byanticipating the vehicle's future movements and location in advance.Hybrid experiences may be provided where portions of the virtual realityenvironment are computed and cached and other information is generatedas it becomes available, either locally by a client device or remotelyby virtual reality generator 121 and electronic horizon generator 131via network 127.

The databases 123 a-n may include a virtual reality imagery databaseincluding multiple virtual environments and virtual elements. Thedatabases 123 a-n may include map databases, predictive navigationdatabases, a road imagery database including street level images, pointcloud data, and/or existing map data. As shown in FIG. 9, a master copyof part or all of database 123 may be stored at the virtual realitygenerator system 121 and/or electronic horizon generator 131. Thedatabases 123 b-n may include alternative versions or past versions ofvirtual reality elements and/or environments using sensor informationfrom vehicle 129 and/or predictive vehicle navigation informationassociated with navigation maps. The master copy of the database 123 maybe the most current or up to date copy of the database. In addition, theclient devices such as mobile device 122, vehicle 129, or virtualreality device 133 may store a local copy of the database 123. In oneexample, the local copy of the database 123 is a full copy of thedatabase, and in another example, the local copy of the database 123 maybe a cached or partial portion of one or more of databases 123 a-n.

The local copy of the database 123 may include data from variousversions of the database 123 a-n. The cached portion may be definedbased on a geographic location of mobile devices 122, vehicles 129, orvirtual reality devices 133 or a user selection made at the clientdevices 122, 129, or 133. The servers 125 may send virtual reality dataand/or predictive navigation data to the client devices 122, 129, or133.

The mobile device 122 may be a personal navigation device (PND), aportable navigation device, smart phone, a mobile phone, a personaldigital assistant (PDA), a car, a tablet computer, a notebook computer,and/or any other known or later developed connected device or personalcomputer. Non-limiting embodiments of navigation devices may alsoinclude relational database service devices, mobile phone devices, orcar navigation devices. Mobile device 122 may be coupled with or mountedon virtual reality device 133, such that a display of the mobile device122 constitutes the display of virtual reality device 133. Mobile device122 may include one or more database 124. The content of database 124may include some or all portions of data stored in databases 123 a-n.

Vehicle 129 may be coupled with a client device or may include one ormore mobile devices 122, one or more virtual reality devices 133, andone or more sensors 126. Vehicle 129 may have a mobile device 122 and/orvirtual reality device 133 associated with one or more passengers.Vehicle 129 may be an autonomous driving vehicle, a data acquisitionvehicle, or a vehicle equipped with navigation or other communicationcapabilities. Sensors 126 may include short range sensors capable ofcollecting data at a range of up to a few hundred meters/feet. Nonlimiting examples of short range sensors include both active and passivedetectors including, but not limited to, infrared detectors, ultrasonicdetectors, microwave detectors, acoustic detectors, piezoelectric,photoelectric, wideband spread-spectrum, inductive loop, position,velocity, acceleration detectors. Some sensors may acquire informationabout the current state of the vehicle including engine pressure,temperature, tire pressure, wheel speed and vibration. Some sensors 126may detect data used by the predictive navigation system including butnot limited to, battery charge level, gas level, turn signal setting,steer angle, gear selection, windshield wiper setting, headlight andinterior light settings, obstacle detection, and a navigation system.Some sensor data may be acquired by other vehicles on the roadway usedto determine anticipated road conditions for vehicles behind it.

The virtual reality device 133 may include virtual reality headsets,helmets, glasses, screens, controllers, remotes, or joysticks. Virtualreality device may further include one or more processor, memory, inputdevice, communication interface, position circuitry and/or display. Thevirtual reality device 133 may be integrated with or coupled with amobile device 122.

The virtual reality generator system 121, electronic horizon generatorsystem 131, the workstation 128, mobile device 122, vehicle 129, virtualreality device 133, networking device 135 and traffic device 137 arecoupled with the network 127. The phrase “coupled with” is defined tomean directly connected to or indirectly connected through one or moreintermediate components. Such intermediate components may includehardware and/or software-based components.

The network 127 may include wired networks, wireless networks, orcombinations thereof. The wireless network may be a cellular telephonenetwork, an 802.11, 802.16, 802.20, or WiMax network. Further, thenetwork 127 may be a public network, such as the Internet, a privatenetwork, such as an intranet, or combinations thereof, and may utilize avariety of networking protocols now available or later developedincluding, but not limited to TCP/IP based networking protocols.

As illustrated in FIG. 6, the system 120 of FIG. 5 may includeadditional devices and components communicating with the network and/orvirtual reality generator 121 and/or electronic horizon generator 131.These additional devices and components may include additional vehicles129, networking device 135, and traffic device 137. Vehicles 129 mayprovide information to virtual reality generator 121 and electronichorizon generator 131 that may be used to generate virtual realitycontent for passengers in other vehicles 129.

Networking devices 135 may include base transceiver stations (BTS)routers, switches, access devices, multiplexers, and servers. Networkingdevices 135, such as a BTS may store data collected by passing vehiclesvia Edge computing. Vehicle data stored by networking devices 135 maylater be accessed or received via network 127 to generate virtualreality content for later passing vehicles. Traffic device 137 includedata sources providing data associated with traffic signals and patternsincluding traffic lights, pedestrian crosswalk lights, railroad crossingarms, traffic cameras, radar sensors, traffic signal preemption devices,or any other traffic or transportation related device. Traffic device137 may provide data associated with roadway infrastructure. Data storedlocally on networking devices 135 may include pre-defined orpre-computed models for the virtual reality system to reduce latency inthe system. Topography, local traffic conditions, infrastructureinformation, or other data may be stored in a networking device 135 suchas a base station. The storage and preparation of such data may provideflexibility for the system, eliminating the need for some information tobe passed through portions of network 127 to virtual reality generator121 or electronic horizon generator 131. Locally stored information maybe received directly by the client devices 122, 129, and/or 133 andprocessed locally on one or more of these devices to produce or enhancevirtual reality elements and environments of the system.

Vehicle 129 may communicate with one or more additional vehicles 129near the current location of vehicle 129. Alternatively or additionally,data originating from additional vehicles 129 received or acquired byvirtual reality generator 121 and/or electronic horizon generator 131over network 127 and may be used to generate predictive navigation.

FIG. 7A illustrates an embodiment of system 120 in which mobile device133 serves as the primary connection to virtual reality generator 121and electronic horizon generator 131. The virtual reality device 133 andvehicle 129 may transmit and receive data to mobile device 122. Mobiledevice 122 may receive virtual reality data from virtual realitygenerator 121 and/or electronic horizon generator 131 and transmit thereceived data to virtual reality device 133 and/or vehicle 129. Mobiledevice 122 may receive or access data from sensor 126 and, in turn,transmit the received sensor data to the electronic horizon generator131 and/or virtual reality generator 121 via network 127.

In some embodiments of system 120 as illustrated in FIG. 7B, the virtualreality device 133 may be connected to the electronic horizon generator131 and the virtual reality generator 121 via network 127. The virtualreality device 133 may be connected both the vehicle 129 and a mobiledevice 122 via a wired or wireless personal area network (PAN) such asBluetooth or Universal Serial Bus (USB), Wi-Fi, sim card, cellularnetwork, or other wired or short range connection. The virtual realitydevice 133 may receive or retrieve sensor related information directlyfrom vehicle 129, or may receive or retrieve sensor related informationassociated with vehicle 129 via network 127. Virtual reality device 133,mobile device 122, and vehicle 129 may be connected to other virtualreality devices 133 and/or mobile devices 122 of passengers in the samevehicle 129. Client devices 122, 133, and 129 may further connect withclient devices associated with other vehicles proximate to vehicle 129.

As illustrated in FIG. 7C, system 120, vehicle 129 may be connected tothe electronic horizon generator 131 and virtual reality generator 121via network 127. Mobile device 122 and virtual reality device 133 may beconnected to vehicle 129. In some cases and/or during some periods oftime, mobile device 122 and/or virtual reality device 133 may not bedirectly connected to network 127 and may send and receive informationassociated with virtual electronic horizon generator 131 and virtualreality generator 121 only through vehicle 129. Unlike FIG. 7B, virtualreality device 133 may not directly access network 127, electronichorizon generator 131, or virtual reality generator 121. Instead,vehicle 129 may provide virtual reality content to virtual realitydevice 133.

FIG. 8 illustrates an exemplary client device such as mobile device 122,vehicle 129, and virtual reality device 133 of the system of thepreceding figures. The client devices 122, 129, and/or 133 include aprocessor 200, a memory 204, an input device 203, a communicationinterface 205, position circuitry 207, and a display 211. Additional,different, or fewer components are possible for the client devices 122,129, and/or 133.

The positioning circuitry 207 may include a Global Positioning System(GPS), Global Navigation Satellite System (GLONASS), or a cellular orsimilar position sensor for providing location data. The positioningsystem may utilize GPS-type technology, a dead reckoning-type system,cellular location, or combinations of these or other systems. Thepositioning circuitry 207 may include suitable sensing devices thatmeasure the traveling distance, speed, direction, and so on, of clientdevices 122, 129, and/or 133. The positioning system may also include areceiver and correlation chip to obtain a GPS signal. Alternatively oradditionally, the one or more detectors or sensors may include anaccelerometer built or embedded into or within the interior of theclient devices 122, 129, and/or 133. The accelerometer is operable todetect, recognize, or measure the rate of change of translational and/orrotational movement of the client devices 122, 129, and/or 133. clientdevices 122, 129, and/or 133 receive location data from the positioningsystem. The location data indicates the location of client devices 122,129, and/or 133.

The input device 203 may be one or more buttons, keypad, keyboard,mouse, stylist pen, trackball, rocker switch, touch pad, voicerecognition circuit, or other device or component for inputting data tothe mobile device 100. The input device 203 and the display 211 may becombined as a touch screen, which may be capacitive or resistive. Thedisplay 211 may be a liquid crystal display (LCD) panel, light emittingdiode (LED) screen, thin film transistor screen, or another type ofdisplay.

The processor 200 and/or processor 300 may include a general processor,digital signal processor, an application specific integrated circuit(ASIC), field programmable gate array (FPGA), analog circuit, digitalcircuit, combinations thereof, or other now known or later developedprocessor. The processor 200 and/or processor 300 may be a single deviceor combinations of devices, such as associated with a network,distributed processing, or cloud computing.

The memory 204 and/or memory 301 may be a volatile memory or anon-volatile memory. The memory 204 and/or memory 301 may include one ormore of a read only memory (ROM), random access memory (RAM), a flashmemory, an electronic erasable program read only memory (EEPROM), orother type of memory. The memory 204 and/or memory 301 may be removablefrom client devices 122, 129, and/or 133, such as a secure digital (SD)memory card.

The communication interface 205 and/or communication interface 305 mayinclude any operable connection. An operable connection may be one inwhich signals, physical communications, and/or logical communicationsmay be sent and/or received. An operable connection may include aphysical interface, an electrical interface, and/or a data interface.The communication interface 205 and/or communication interface 305provides for wireless and/or wired communications in any now known orlater developed format.

Client devices 122, 129, and/or 133 are configured to execute algorithmsto for generating, enhancing, rendering, and/or displaying virtualreality elements and/or virtual reality environments. Virtual realityelements may be dependent on information acquired by sensors of vehicle129, using predictive vehicle navigation associated with mappingdatabases, and may also be dependent on information originating fromother sources such as networking devices 135, traffic signals 137,emergency services and the like. Client devices 122, 129, and/or 133 maybe configured to acquire sensor information from one or more vehicles129 including geolocation, short range sensors of vehicles 129, and/ornavigation information and/or predictive vehicle navigation informationfrom electronic horizon generator 131. Using acquired information, theclient devices 122, 129, and/or 133 independent of, or in conjunctionwith, servers 125 generate, enhance, render, and/or display virtualreality elements and/or virtual reality environments.

Using input from the end user, client devices 122, 129 and/or 133, theclient devices and/or electronic horizon generator 131 may examinepotential routes between an origin location and a destination locationto determine an optimum route. Client devices 122, 129 and/or 133, theclient devices and/or electronic horizon generator 131 may then providethe end user with information about the optimum route in the form ofguidance that identifies the maneuvers required to be taken by the enduser to travel from the origin to the destination location. Some clientdevices 122, 129 and/or 133, may additionally display detailed maps ondisplays outlining the route, the types of maneuvers to be taken atvarious locations along the route, locations of certain types offeatures, and so on.

FIG. 9 illustrates example servers such as the servers 125 of virtualreality generator 121 and/or electronic horizon generator 131. Theserver 125 includes a processor 300, a communication interface 305, anda memory 301. The server 125 may be coupled to one or more databases 123and a workstation 128. The workstation 128 may be used to enter dataregarding sensor information, images, map data, virtual realityelements, virtual reality environments, navigation information, andpredictive navigation information. The databases 123 may includeinformation entered from workstation 128, sensor information, images,map data, virtual reality elements, virtual reality environments,navigation information, and predictive navigation information.Additional, different, or fewer components may be provided in the server125. FIG. 1 illustrates an example flow diagram for the operation ofserver 125. Additional, different, or fewer acts may be provided.

The processor 300 and/or processor 200 may include a general processor,digital signal processor, an application specific integrated circuit(ASIC), field programmable gate array (FPGA), analog circuit, digitalcircuit, combinations thereof, or other now known or later developedprocessor. The processor 200 and/or processor 300 may be a single deviceor combinations of devices, such as associated with a network,distributed processing, or cloud computing.

The memory 301 and/or memory 204 may be a volatile memory or anon-volatile memory. The memory 301 and/or memory 204 may include one ormore of a read only memory (ROM), random access memory (RAM), a flashmemory, an electronic erasable program read only memory (EEPROM), orother type of memory. The memory 301 and/or memory 204 may be removablefrom client devices 122, 129, and/or 133, such as a secure digital (SD)memory card.

The communication interface 305 and/or communication interface 205 mayinclude any operable connection. An operable connection may be one inwhich signals, physical communications, and/or logical communicationsmay be sent and/or received. An operable connection may include aphysical interface, an electrical interface, and/or a data interface.The communication interface 205 and/or communication interface 305provides for wireless and/or wired communications in any now known orlater developed format.

The term “computer-readable medium” includes a single medium or multiplemedia, such as a centralized or distributed database, and/or associatedcaches and servers that store one or more sets of instructions. The term“computer-readable medium” shall also include any medium that is capableof storing, encoding or carrying a set of instructions for execution bya processor or that cause a computer system to perform any one or moreof the methods or operations disclosed herein.

In a particular non-limiting, exemplary embodiment, thecomputer-readable medium can include a solid-state memory such as amemory card or other package that houses one or more non-volatileread-only memories. Further, the computer-readable medium can be arandom access memory or other volatile re-writable memory. Additionally,the computer-readable medium can include a magneto-optical or opticalmedium, such as a disk or tapes or other storage device to capturecarrier wave signals such as a signal communicated over a transmissionmedium. A digital file attachment to an e-mail or other self-containedinformation archive or set of archives may be considered a distributionmedium that is a tangible storage medium. Accordingly, the disclosure isconsidered to include any one or more of a computer-readable medium or adistribution medium and other equivalents and successor media, in whichdata or instructions may be stored. These examples may be collectivelyreferred to as a non-transitory computer readable medium.

In an alternative embodiment, dedicated hardware implementations, suchas application specific integrated circuits, programmable logic arrays,and other hardware devices can be constructed to implement one or moreof the methods described herein. Applications that may include theapparatus and systems of various embodiments can broadly include avariety of electronic and computer systems. One or more embodimentsdescribed herein may implement functions using two or more specificinterconnected hardware modules or devices with related control and datasignals that can be communicated between and through the modules, or asportions of an application-specific integrated circuit. Accordingly, thepresent system encompasses software, firmware, and hardwareimplementations.

In accordance with various embodiments of the present disclosure, themethods described herein may be implemented by software programsexecutable by a computer system. Further, in an exemplary, non-limitedembodiment, implementations can include distributed processing,component/object distributed processing, and parallel processing.Alternatively, virtual computer system processing can be constructed toimplement one or more of the methods or functionality as describedherein.

Although the present specification describes components and functionsthat may be implemented in particular embodiments with reference toparticular standards and protocols, the invention is not limited to suchstandards and protocols. For example, standards for Internet and otherpacket switched network transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP,HTTPS) represent examples of the state of the art. Such standards areperiodically superseded by faster or more efficient equivalents havingessentially the same functions. Accordingly, replacement standards andprotocols having the same or similar functions as those disclosed hereinare considered equivalents thereof.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, and it can bedeployed in any form, including as a standalone program or as a module,component, subroutine, or other unit suitable for use in a computingenvironment. A computer program does not necessarily correspond to afile in a file system. A program can be stored in a portion of a filethat holds other programs or data (e.g., one or more scripts stored in amarkup language document), in a single file dedicated to the program inquestion, or in multiple coordinated files (e.g., files that store oneor more modules, sub programs, or portions of code). A computer programcan be deployed to be executed on one computer or on multiple computersthat are located at one site or distributed across multiple sites andinterconnected by a communication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., a field programmable gate array (FPGA) or anapplication specific integrated circuit (ASIC).

As used in this application, the term ‘circuitry’ or ‘circuit’ refers toall of the following: (a) hardware-only circuit implementations (such asimplementations in only analog and/or digital circuitry) and (b) tocombinations of circuits and software (and/or firmware), such as (asapplicable): (i) to a combination of processor(s) or (ii) to portions ofprocessor(s)/software (including digital signal processor(s)), software,and memory(ies) that work together to cause an apparatus, such as amobile phone or server, to perform various functions) and (c) tocircuits, such as a microprocessor(s) or a portion of amicroprocessor(s), that require software or firmware for operation, evenif the software or firmware is not physically present.

This definition of ‘circuitry’ applies to all uses of this term in thisapplication, including in any claims. As a further example, as used inthis application, the term “circuitry” would also cover animplementation of merely a processor (or multiple processors) or portionof a processor and its (or their) accompanying software and/or firmware.The term “circuitry” would also cover, for example and if applicable tothe particular claim element, a baseband integrated circuit orapplications processor integrated circuit for a mobile phone or asimilar integrated circuit in server, a cellular network device, orother network device.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor receives instructions and data from a read only memory or arandom access memory or both. The essential elements of a computer are aprocessor for performing instructions and one or more memory devices forstoring instructions and data. Generally, a computer also includes, oris operatively coupled to receive data from or transfer data to, orboth, one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio player, a Global Positioning System (GPS) receiver, to namejust a few. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a devicehaving a display, e.g., a cathode ray tube (CRT) or liquid crystaldisplay (LCD) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

The illustrations of the embodiments described herein are intended toprovide a general understanding of the structure of the variousembodiments. The illustrations are not intended to serve as a completedescription of all of the elements and features of apparatus and systemsthat utilize the structures or methods described herein. Many otherembodiments may be apparent to those of skill in the art upon reviewingthe disclosure. Other embodiments may be utilized and derived from thedisclosure, such that structural and logical substitutions and changesmay be made without departing from the scope of the disclosure.Additionally, the illustrations are merely representational and may notbe drawn to scale. Certain proportions within the illustrations may beexaggerated, while other proportions may be minimized. Accordingly, thedisclosure and the figures are to be regarded as illustrative ratherthan restrictive.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the invention or of what may beclaimed, but rather as descriptions of features specific to particularembodiments of the invention. Certain features that are described inthis specification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable sub-combination. Moreover, although features may be describedabove as acting in certain combinations and even initially claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a sub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings and describedherein in a particular order, this should not be understood as requiringthat such operations be performed in the particular order shown or insequential order, or that all illustrated operations be performed, toachieve desirable results. In certain circumstances, multitasking andparallel processing may be advantageous. Moreover, the separation ofvarious system components in the embodiments described above should notbe understood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

One or more embodiments of the disclosure may be referred to herein,individually and/or collectively, by the term “invention” merely forconvenience and without intending to voluntarily limit the scope of thisapplication to any particular invention or inventive concept. Moreover,although specific embodiments have been illustrated and describedherein, it should be appreciated that any subsequent arrangementdesigned to achieve the same or similar purpose may be substituted forthe specific embodiments shown. This disclosure is intended to cover anyand all subsequent adaptations or variations of various embodiments.Combinations of the above embodiments, and other embodiments notspecifically described herein, will be apparent to those of skill in theart upon reviewing the description.

The Abstract of the Disclosure is provided to comply with 37 C.F.R.§1.72(b) and is submitted with the understanding that it will not beused to interpret or limit the scope or meaning of the claims. Inaddition, in the foregoing Detailed Description, various features may begrouped together or described in a single embodiment for the purpose ofstreamlining the disclosure. This disclosure is not to be interpreted asreflecting an intention that the claimed embodiments require morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive subject matter may be directed toless than all of the features of any of the disclosed embodiments. Thus,the following claims are incorporated into the Detailed Description,with each claim standing on its own as defining separately claimedsubject matter.

It is intended that the foregoing detailed description be regarded asillustrative rather than limiting and that it is understood that thefollowing claims including all equivalents are intended to define thescope of the invention. The claims should not be read as limited to thedescribed order or elements unless stated to that effect. Therefore, allembodiments that come within the scope and spirit of the followingclaims and equivalents thereto are claimed as the invention.

We claim:
 1. A method comprising: receiving a vehicle state sensor datarecord associated with a current state of a vehicle; receiving apredictive vehicle navigation data record based on map data, the mapdata associated with a current location of the vehicle; adapting a firstelement of a virtual reality environment based on the received vehiclestate sensor data record; and generating a second element of the virtualreality environment based on the received predictive vehicle navigationdata record.
 2. The method of claim 1, the method further comprising:receiving a vehicle environment sensor data record associated withcurrent surroundings of the vehicle; and generating a third element ofthe virtual reality environment based on the received vehicleenvironment sensor data record associated with the current surroundingsof the vehicle.
 3. The method of claim 1, the method further comprising:sending an instruction to adjust an instrument of the vehicle based onthe virtual reality environment.
 4. The method of claim 1, the methodfurther comprising: receiving an externally generated vehicle statesensor data record from at least one other vehicle; and adapting afourth element of the virtual reality environment based on the receivedexternally generated vehicle sensor data record.
 5. The method of claim1, the method further comprising: receiving an infrastructure datarecord from roadway infrastructure; and adapting a fifth element of thevirtual reality environment based on the received infrastructure datarecord.
 6. The method of claim 1, wherein the received vehicle statesensor data record includes a current state of acceleration ordeceleration of the vehicle.
 7. The method of claim 1, wherein thevirtual reality environment includes a path corresponding to a predictedroute of the vehicle.
 8. An apparatus comprising: at least oneprocessor; and at least one memory including computer program code forone or more programs; the at least one memory and the computer programcode configured to, with the at least one processor, cause the system toat least perform: receive a vehicle state sensor data record associatedwith a current state of a vehicle; receive a predictive vehiclenavigation data record based on map data, the map data associated with acurrent location of the vehicle; receive a vehicle environment sensordata record associated with current surroundings of the vehicle; adapt afirst element of a virtual reality environment based on the receivedvehicle state sensor data records; generate a second element of thevirtual reality environment based on the received predictive vehiclenavigation data record; and generate a third element of the virtualreality environment based on the received vehicle environment sensordata record associated with the current surroundings of the vehicle. 9.The system of claim 8, the at least one memory and the computer programcode configured to, with the at least one processor, cause the system toat least perform: send an instruction to adjust an instrument of thevehicle based on the virtual reality environment.
 10. The system ofclaim 8, the at least one memory and the computer program codeconfigured to, with the at least one processor, cause the system to atleast perform: receive an externally generated vehicle state sensor datarecord from at least one other vehicle; and adapt a fourth element ofthe virtual reality environment based on the received externallygenerated vehicle sensor data record.
 11. The system of claim 8, the atleast one memory and the computer program code configured to, with theat least one processor, cause the system to at least perform: receive aninfrastructure data record from roadway infrastructure; and adapt afifth element of the virtual reality environment based on the receivedinfrastructure data record.
 12. The system of claim 8, wherein thereceived vehicle state sensor data record includes a current state ofacceleration or deceleration of the vehicle.
 13. The system of claim 8,wherein the virtual reality environment includes a path corresponding toa predicted route of the vehicle.
 14. A non-transitory computer readablemedium including instructions that when executed are operable to:receive a vehicle state sensor data record associated with a currentstate of a vehicle; receive a predictive vehicle navigation data recordbased on map data, the map data associated with a current location ofthe vehicle; receive a vehicle environment sensor data record associatedwith current surroundings of the vehicle; adapt a first element of avirtual reality environment based on the received vehicle state sensordata records; generate a second element of the virtual realityenvironment based on the received predictive vehicle navigation datarecord; and generate a third element of the virtual reality environmentbased on the received vehicle environment sensor data record associatedwith the current surroundings of the vehicle.
 15. The non-transitorycomputer readable medium of claim 14, including instructions that whenexecuted are operable to: send an instruction to adjust an instrument ofthe vehicle based on the virtual reality environment.
 16. Thenon-transitory computer readable medium of claim 14, includinginstructions that when executed are operable to: receive an externallygenerated vehicle sensor data record from at least one other vehicle;and adapt a fourth element of the virtual reality environment based onthe received externally generated vehicle sensor data record.
 17. Thenon-transitory computer readable medium of claim 14, includinginstructions that when executed are operable to: receive aninfrastructure data record from roadway infrastructure; and adapt afifth element of the virtual reality environment based on the receivedinfrastructure data record.
 18. The non-transitory computer readablemedium of claim 14, wherein the received vehicle state sensor datarecord includes a current state of acceleration or deceleration of thevehicle.
 19. The non-transitory computer readable medium of claim 14,wherein the generated the virtual reality environment includes a pathcorresponding to a predicted route of the vehicle.
 20. Thenon-transitory computer readable medium of claim 14, includinginstructions that when executed are operable to: receive an autonomousdriving vehicle data record associated with driver control: and generatea sixth element of the virtual reality environment based on the receivedautonomous driving vehicle data record.